1. Field of the Invention
The invention relates to means by which mobile robots, equipped with computer operated onboard internet web servers and short range digital radio transceivers may be remotely teleoperated over the internet, and directed to interact with other external computer controlled devices also equipped with short range digital radio transceivers.
2. Description of the Related Art
Although self-propelled mobile robots have been the subject of speculative literature and experimentation for a number of years, the field is still in its infancy. With the partial exception of industrial automation for highly repetitive situations, educational purposes, and toys, mobile robots are not widely used. This is because the problems of coping with machine vision, artificial intelligence, general purpose robotic xe2x80x9carmxe2x80x9d and xe2x80x9chandxe2x80x9d like actuation systems, limited battery life, and the like make it difficult to devise flexible and general purpose systems suitable for widespread use.
Because robotic artificial intelligence is particularly difficult to achieve, human operators often remotely control robots. A number of techniques for robotic remote control (teleoperation) are known in the art, and these are discussed in the book: xe2x80x9cRemote Control Roboticsxe2x80x9d by Craig Sayers, published by Springer-Verlag, New York, (1998) and incorporated herein by reference.
Other prior art includes U.S. Pat. No. 4,964,062 which teaches a type of robotic arm that may be controlled by radio from a remote location, and may be programmed by radio from a remote location. U.S. Pat. No. 5,331,413 describes an external camera used to view a robot. U.S. Pat. Nos. 5,231,693 and 5,046,022 describe methods of semi-automating telerobotic control. U.S. Pat. No. 6,144,844 describes methods to compensate for variable teleoperation delay.
Recently, programming techniques used to increase the power and flexibility of the Internet have matured, and a number of these newer programming techniques can also be adapted to robotic control methods. In particular, this includes programming languages, interfaces, and protocols used to generate (serve) and interpret (browse) world wide web pages on the Internet.
World wide web (also called xe2x80x9cWebxe2x80x9d or xe2x80x9cNetxe2x80x9d) servers are computer programs, such as xe2x80x9cApachexe2x80x9d, and the like, that communicate over the internet in the form of variants of Standard Generalized Markup Language (SGML), such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), Extensible HTML (XHTML), and the like. These servers accept commands from a remote user, and respond by returning SGML variant xe2x80x9cweb pagesxe2x80x9d or forms which can be read and interpreted by the remote user""s browser.
For purposes of this discussion, xe2x80x9cweb pagexe2x80x9d is defined as an HTML, XML, or XTML electronic document on the world wide web, identified by a unique Universal Resource Identifier (URI) or unique Universal Resource Locator (URL), and transmitted over the internet using the (HTTP) Hypertext Transfer Protocol, and usually the TCP/IP protocol.
Also for purposes of this discussion, xe2x80x9cweb serverxe2x80x9d is defined as computer program functionality (either stand alone, or part of an operating system) that, in response to an internet URL or URI request, sends back a xe2x80x9cweb pagexe2x80x9d file to the internet address that made the request, or alternatively passes the URL or URI request through a xe2x80x9cCommon Gateway Interfacexe2x80x9d or CGI protocol to a CGI compliant program, and when the CGI compliant program has output, and passes the output back to the internet address that made the original URL or URI request.
Use of the CGI interface greatly enhances the flexibility and power of web/serverxe2x80x94SGML variant web form technology. Here, the HTML or SMGL variant web form can be instructed to pass data of any sort to and from the user to programs, executable scripts, or software processes that are distinct (external) from the main web server program. This non-web server software may be a database, image manipulation program, robot control program, or any other independently running process. Data submitted from a web browser to a web server""s CGI interface is generally submitted either appended to a CGI URL address (GET method, limited to 255 maximum characters of data) or through a STDIN standard input method (POST method, no data size limitation). Data from a web server to a web browser may be of any sort, as long as it complies with the Multipurpose Internet Mail Extensions (MIME) format.
Prior art on the GCI interface includes U.S. Pat. Nos. 6,041,362; 5,961,594; 5,905,908; and 5,742,845. A more complete CGI discussion is given in Dwight, Erwin and Niles book: xe2x80x9cUsing CGI, Second Editionxe2x80x9d, 1997, published by Que corporation, and incorporated herein by reference.
In addition to SGML variant files (HTML, XML, XHTML, etc.), programming languages such as Java are also frequently transmitted by servers on the word wide web. Java (and Java-like languages such as C#), is a general purpose, object oriented, high level programming language, commonly used on the world wide web to deliver enhanced functionality to web pages. Typically Java commands are embedded in SGML variant files that frequently sent out (and occasionally also received) by web servers. Such Java xe2x80x9cappletsxe2x80x9d can then perform many specialized functions. In particular, Java applets may be used to extend the basic SGML functionality, and can translate user commands to other languages, display video, display interactive graphics, and other useful functions.
Web browsers are likewise commonly known in the art. For the purposes of this patent, a xe2x80x9cweb browserxe2x80x9d is considered to be a program that can read and interpret the SGML variant files sent out by web servers, (primarily HTML files) and optionally execute Java and/or additional plug-in module commands. Examples of modern web browsers include Microsoft Internet Explorer 5.0 or greater, Netscape Navigator 5.0 or greater, and the like.
For robotics control purposes, in order to take advantage of the wide variety of xe2x80x9coff-the-shelfxe2x80x9d software, originally written for internet or other purposes, for robotics control, it is preferable to use software (programs) that run on standard operating systems, rather than software that attempts to run directly on the underlying robotic computer hardware. Standard operating systems, such as Windows, Linux, Unix, and the like, manage much of the software burden of a computational system by controlling access to underlying hardware, memory management, and the like, relieving the programmer of much complexity. Popular (e.g. widely used) multi-tasking operating systems are particularly advantageous, since thousands of programs are available that have previously been written for them. By use of a popular operating system, a robotics programmer may leverage his or her limited time and energy by using standard programs to handle most of the robotics system, and only writing robot specific software when absolutely necessary. The one drawback of this approach from a robotics control standpoint is some potential loss or degradation of precise real-time determinism and control, which can be overcome by the use of real-time modifications or patches to such operating systems. An example of a popular operating system (Linux), modified to enhance real-time determinism and control, is taught by U.S. Pat. No. 5,995,745 for xe2x80x9creal time Linuxxe2x80x9d. Other such real-time operating system enhancements also exist.
Previous art teaches that both mobile and non-mobile robots may have their movement and video functions controlled remotely by using web browser/web server technology. For example, the Intelligent Autonomous Systems Engineering Laboratory of the University of the West of England, Bristol teaches that web server technology may be used in mobile robotic systems. They have developed the xe2x80x9cLinuxBotxe2x80x9d. This is a mobile robot, intended as an educational project, that performs a number of functions, including movement and wall avoidance, as well as serving web pages, using an on-board multitasking embedded Linux operating system environment with a wireless link to a local area network. The robot has no xe2x80x9carmsxe2x80x9d or other actuators, nor other means to control other external computer controlled devices. Its primary purpose is as an academic training device.
iRobot corporation of Somerville, Mass. produces the xe2x80x9ciRobot-LExe2x80x9d, a mobile robot containing an onboard computer running an xe2x80x9capachexe2x80x9d web server under the Linux operating system. Users may remotely control this system over the Internet using standard web browsers. Because the device contains neither robotic xe2x80x9carmsxe2x80x9d other actuators, or means to control other external computer controlled devices, it has limited practical utility.
A number of examples of non-mobile robotic systems, such as robotic arms controlled by remote web browsers over the Internet, also exist. Some of these are documented in Chapter 4 of the previously discussed xe2x80x9cRemote Control Roboticsxe2x80x9d reference. Typically these are demonstration systems that can move a few test items, or perform some other limited chore. Other examples include different types of robotic telescopes. In one implementation, a remote viewer using a standard Internet browser positions a robot telescope. The telescope is commanded by a version of XML called AMIL (Astronomical Instrument Markup Language), which is interpreted by a Java program running locally on the robotic telescope.
Although such prior art is useful for situations in which self-mobile robots are used to directly interact with their local environment, or in which peripheral devices attached to non-mobile robots are remotely controlled over the internet, there exist situations in which it is desirable to extend the functionality of a web browser/server controlled mobile robotic system beyond those peripherals that are directly connected to the robot. In particular, there are situations where it is desirable to bring a capable mobile robot close to a particular location, use the sensors onboard the robot to observe local conditions, and then manipulate one or more additional devices in the location that would otherwise be impractical to control by direct manipulation by the robots onboard arms, manipulators, etc.
For example, there may be situations where it is desirable to employ a mobile robot for security guard functions. Here, the robot may be given a general route to travel by a remote Internet operator, as well as standing orders to observe its surroundings by its onboard camera, and as check and adjust the status of various external robotic cooperative monitors. The remote internet operator may also wish to leave standing orders for the robot to travel different routes, open robotic cooperative doors, sound robotic cooperative alarms, etc., depending upon the data that is received from the external monitors.
As a second example, there may be situations where it is desirable to employ a mobile robot to monitor and adjust robotic cooperative equipment in an industrial or laboratory facility. Here, the robot would be given general routes to travel by one or more workers over the Internet. The robot would travel the specified routes, check the status of external equipment monitors, and additionally modify the state of the external equipment, using various standing orders that the remote operators may have left. Such a system would be particularly useful in a flexible manufacturing industrial setting, or laboratory, where many operators, residing in many remote locations, have a desire to use the equipment on a time-share basis.
In a third example, the mobile robot of this invention might be used in a home situation to travel throughout the home and observe pets, children, or elderly people, and perform service functions such as fetching food from specially modified robotic cooperative refrigerators, cook food in robotic cooperative ovens, or process utensils or dishes in specially modified, robotic cooperative, dishwashers, garbage compactors, etc. Other home functions where such a robot could interact with specially modified, robotic cooperative, appliances include sprinklers to water plants, equipment to care for pets, cleaning equipment, heating and air conditioning equipment, security equipment, and the like.
For the purposes of this discussion, a xe2x80x9cmobilexe2x80x9d robot is considered to be a robot that moves on solid surfaces, water, or air due to the controlled action of onboard motors or mechanical actuators, such as motorized wheels, tracks, legs, propellers, jets, and the like. A robot that moves only by the application of outside force, even if portable and easily carried, is not considered to be xe2x80x9cmobilexe2x80x9d.
Because it is difficult to design a mobile robot with sufficient visual acuity or dexterity to work with a wide variety of human operated equipment, the problem may be considerably simplified if the requirement that the robot directly observe external sensors or directly manipulate external equipment is reduced or dropped. This may be done by modifying the external sensors or equipment to be xe2x80x9crobotic cooperativexe2x80x9d by having an ability to report their status, and be operated by, short range, bi-directional, digital radio links (abbreviated here as xe2x80x9cSBDRLxe2x80x9d for Short-range Bi-directional Digital Radio Link). If this is done, then the robot need only get close enough to the sensor or equipment to make radio contact with the external device. Once this is done, the robot may then ascertain the other device""s status by short-range digital radio link, and optionally make additional observations or manipulations using the robot""s own on-board equipment. The robot can then transmit its findings and receive commands from operators from any location throughout the world using inexpensive and commonly used Internet (web) browser systems to relay to the external device.
To achieve these objectives, it is desirable to give a web browser/server controlled mobile robot an additional capability to manipulate its local environment by sending and receiving bi-directional short-range digital radio signals that can be used to control local devices that are external to the robot. In this context, xe2x80x9clocalxe2x80x9d and xe2x80x9cshort-rangexe2x80x9d are defined to be distances between about 0 and 300 feet.
Means to control various computerized devices by local digital radio links are known in the art. Although many different radio frequencies and digital encoding schemes can be used for these purposes, use of the 2.4 gigahertz (gHz) industrial-scientific-medical frequency band, which is internationally reserved for local digital links, and does not require a license, is particularly advantageous. It is also advantageous to use standard digital-encoding schemes, as this increases the variety of devices that the robot can communicate with.
There are a number of standard protocols for communicating digital information in the 2.4 gHz frequency, such as the Institute of Electrical and Electronics Engineers (IEEE) standards 802.11, and the IEEE 802.15 and 802.15b standard for personal area networks (PAN). The disclosures for these standards are incorporated herein by reference.
IEEE standard 802.11 discloses a set of local digital radio link protocols compatible with the HomeRF specification. The HomeRF standard, in conjunction with the Shared Wireless Access Protocol (SWAP), enables local low power digital 2.4 gigahertz radio communication with a range of up to 150 feet between devices. Up to 10 different devices may be networked within this radius.
IEEE standard 802.15 discloses a set of local digital radio link protocols compatible with the Bluetooth(trademark) specification. (Bluetooth is a trademark of the Bluetooth special interest group). Bluetooth compliant low power digital radio systems can automatically establish very short range (0-30 feet) or longer short range (0-300 feet) wireless digital radio linkages with other Bluetooth compliant devices. Further discussion of the Bluetooth standard may be found in the book: xe2x80x9cBluetooth revealedxe2x80x9d by Miller and Bisdikian, Prentice Hall PTR publisher (2000), and incorporated herein by reference.
Previous examples of remotely controlled robotic systems that use short-range digital radio links to control local peripherals include the 1996-1997 NASA Mars xe2x80x9cPathfinderxe2x80x9d and xe2x80x9cSojournerxe2x80x9d robotic system. In this system a stationary robot (the Pathfinder) was controlled by unique NASA protocol using a long-range radio link to earth. The stationary Pathfinder robotic system then established a short-range digital link at 459 MHz with the mobile Sojourner robotic rover, and controlled its movement and sensors. The results were communicated back to earth using unique NASA transmission schemes. Because these techniques used nonstandard, non-SGML variant, non-HTTP protocols, complex, customized and expensive equipment and programs were required in order to correctly control the Sojourner robotic rover. The techniques and software used were not general purpose, and thus could not be used for purposes beyond controlling the specific system at hand.
There appears to no prior art teaching the use of mobile, Internet capable, web browser/server controlled robots to travel to nearby devices, and subsequently monitor and/or control the devices by use of bi-directional short-range digital radio links. Nonetheless, such devices would enable remote users to control sophisticated robotic systems using inexpensive and readily available equipment and software, and thus would be of significant practical utility.
At present, mobile robots and robotic systems tend to be inflexible, specialized, and expensive. In order to bring the advantages of robotic automation into more general use, robots must be designed that are simple, low-cost, and general purpose. Such robotic systems can be can be constructed according to the principles disclosed herein. This is done by combining the flexibility and generality of Internet technology, with mobile, capable, and general-purpose robots, and multiple SBDRL external peripherals that are optimized to address specific tasks.
In the broadest sense of the invention, the xe2x80x9crobotxe2x80x9d disclosed here is a self-propelled mobile internet server with a capability of establishing both a first connection to the internet for robotic control purposes, and a capability of establishing a second bi-directional digital radio connection to one or more local digital radio devices that are not themselves directly connected to the internet, and which have a maximum radio range of about 300 feet. In a preferred embodiment, this short-range wireless digital connection will use the 2.4 GHz band and digital protocols following the IEEE 802.11, 802.15, or other digital communications protocol. By employing the proper set of external short-range digital radio devices capable of interfacing with the robot, a system of considerable practical utility can be devised.
It is contemplated that the self-propelled mobile Internet server robot will generally serve SGML variant web pages that are designed to be read by humans, and work with standard Internet web browsers under direct human control. However for greatest generality and usefulness, direct human control is not strictly required. Human readable SGML variant (typically HTML) web pages can also be scanned, parsed, and interpreted by automated techniques. Depending upon the situation at hand, it may be advantageous to parse the web pages automatically at a remote site, and remotely control the robot by remote computational resources, (standard computer microprocessors, supercomputers, computer clusters, and the like) that are more sophisticated than the robot""s own onboard computer.
The robotic system disclosed here may be controlled by three different general methods. These control methods are local robotic control, remote automated control, and remote human control. Local robotic control would typically handle routine, simple, or fast response time tasks that are best done by executing previously stored instructions using the robot""s onboard computing facilities. Remote automated control would typically consist of standardized but more computationally challenging tasks, such as sophisticated computer vision, sophisticated robotic control algorithms, etc., best handled by larger and more sophisticated remote computational resources specialized for these purposes. Rarely occurring (non-routine) tasks, complex tasks, or other tasks requiring human judgement would then be handled by humans using remote internet browsers that read standard HTML or SGML variant web pages served by the robot""s onboard web server.
Allowing either remote human and remote computer judgement to be used depending upon the situation increases the scope and flexibility of the robotic system. This is desirable because it enables large numbers of such systems to mass produced for a variety of different uses, leading to a lower per-unit cost for each system.
It is anticipated that the robot would receive its most general (highest level) commands from the remote Internet link. However there may often be situations where it is advantageous to enable the robot to interpret and modify these top-level commands in a quick, real-time manner, depending upon the state of the robot""s sensors and local conditions. In other situations, the robot may be used to perform repetitious tasks that fit within the capability of the robot""s onboard computer processor.
For example, a robot may be given a remote command over the Internet to move forward a number of feet. However if a contact sensor on the robot shows that the robot has encountered an unseen obstruction, the top-level internet command may be modified by contingency software running on the robot, and the robot""s path may be modified or the robot stopped. As another example, a robot may be given an Internet command to grasp an object. The robot may interpret this command by use of its local on-board vision processing to ensure that its mechanical manipulators are properly positioned, etc. In still another example, a robot may simply move over a preprogrammed path in a repetitious manner.
For greatest flexibility, the robotic software should be designed to enable the remote Internet controller to upload new automatic instructions or scripts to the robot. These uploaded automatic commands will typically be used to control the robot when the remote user is off-line, when extremely rapid action is required, or when a complex series of actions is required. In addition to robotic control instructions, these instructions may also include an automatic series of commands to query SBDRL devices, and commands to send to various SBDRL devices.
These local automatic robotic control instructions (or scripts) may be written in a wide variety of formats and languages. These may include anything from simple text commands, to complex binary programs. Since many such local control instructions are possible, each potentially quite situation specific, automatic control instructions written in formats that facilitate rapid development and deployment, preferably by automated means, are preferable. Examples of preferred formats for rapid development and deployment instructions include XML scripts, PERL scripts, JAVA applets, and the like.
In addition to flexible remote control methodology, it is also desirable to give a robot flexible means to interact with its local environment. Here, some deviations from prior art are advantageous.
Prior art generally teaches that mobile robots primarily interact with their environment through sensors and mechanical actuators (robotic arms, and the like) located on the robot itself. However this is not always desirable. Using onboard mechanical actuators as the sole means to interact with the robot""s environment can add significant cost and weight to the robot. Such mechanical systems can also make considerable power demands on a robot""s onboard battery, which will typically have only limited power capacity. An additional problem is that it is difficult to design on-board robotic mechanical manipulators that can function well in a wide range of situations. Robot design constraints, such as limited size and battery life, may require that the robot""s onboard mechanical manipulators have limited dexterity or lifting power.
To enhance flexibility and generality, it is advantageous if the mobile robot is designed with general means to interact with various external sensors, external mechanical manipulators and external appliances. The robot may then travel into range of the external devices, and call upon them to assist the robot at the task at hand. For example, a family of external, fixed position, mechanical manipulators, controlled by a local SBDRL link, can be used in combination with the Internet connected mobile robot to accomplish various tasks. Here, an external, non-mobile set of arms could load and unload items from a carrying tray on the robot, and the like. The robot can use its onboard sensors to monitor the progress of the operation, and to relay commands from the remote Internet operator to the external mechanical manipulators.
In addition to external mechanical manipulators, it is also advantageous if the mobile robot is designed to use SBDRL means to interact with local appliances. Examples of appliances which might be upgraded to add SBDRL capability for these purposes include industrial and laboratory processing equipment, as well as household appliances such as refrigerators, ovens, dishwashers, washing machines, clothes dryers, lights, sprinklers, floor cleaners, lawnmowers, security system, audiovisual systems, pet feeders, heating/air conditioning systems, automatic doors, and the like.
To enhance flexibility and generality, it is also advantageous if the mobile robot is designed with general means to interact with various external sensors. This may include SBDRL capable local vision systems, and other local sensor systems.
Vision systems: Although it is advantageous for a mobile robot to have its own onboard vision system, which can relay its activities and position to remote human or computer control systems, such on-board vision systems suffer from a limited field of view. This can make it particularly difficult for remote human or automated vision systems to control the robot. This problem can be alleviated by the use of external SBDRL vision systems. Such systems can be mounted in more advantageous places, such as the ceiling of a room, which can give a wider field of view. Alternatively, such systems can be mounted in such a way; such as with a microscope attachment, that can give the robot a more detailed field of view.
As an example, a robotic system moving into a room that has a SBDRL vision system on the ceiling will receive a more global perspective of its surroundings. The robot can then internally process this new perspective, or alternatively transmit the new perspective to a remote Internet control site, to assist in robotic control.
Many other types of external sensors are also be linked to a mobile robot by short-range digital links. These include environmental sensors, process sensors, chemical sensors, and the like.
Discovery of locally available SBDRL devices: Although in some situations, the distribution and functionality of the various external SBDRL devices will be known to the robot or the remote operator before the robot moves into proximity of the device, in other situations, this will not be previously known. In these later situations, where the distribution and functionality of the SBDRL devices is not previously known in advance, it is advantageous if the robot and the SBDRL devices are equipped to enable automatic discovery of the nature, functions, and services provided by each device.
A number of schemes to enable automatic discovery of the nature, functions, and services of remote devices are known in the art. This includes schemes such as xe2x80x9cJini(trademark)xe2x80x9d, Universal plug and play(trademark), Salutation(trademark), Service Location protocol, etc. One scheme, particularly useful for xe2x80x9cBluetoothxe2x80x9d (IEEE 802.15b) type devices makes use of the xe2x80x9cService Discovery Protocolxe2x80x9d. Briefly, Bluetooth type SBDRL devices may maintain a service registry consisting of both device identifiers, and functions of the device that are available for use over the SBDRL link. Upon SBDRL query, if the device has its"" discovery functionality enabled, this information is made available to the inquiring device.
By querying such SBDRL service registry information, and either processing it using the robot""s own onboard computer processor(s), or passing the information along to a remote controller on the internet, SBDRL devices not previously known to the robot and/or its remote internet operator may be utilized. This greatly increases the flexibility of the robot, and thus use of SBDRL devices that enable remote discovery of their nature and services is generally preferred. To better facilitate cooperation with other external devices, the robot itself may also maintain a discoverable SBDRL xe2x80x9cservice discovery protocolxe2x80x9d describing the robot""s own nature and useable functions.